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ABSTRACT 

Evaluation practices used by state legislatures to 
monitor educational reform are analyzed in this paper, which focuses 
on the link between politics and choice of oversight strategies and 
evaluation methods. Two oversight strategies are 
compared— "police-patrol" and "fire-alarm." "Fire-alarm" oversight 
involves selective monitoring, triggered by complaints from citizens 
and interest groups who bring potential problems to legislators* 
attention. In "policy-patrol" oversight, legislators monitor policies 
to detect problems during implementation. A total of 57 interviews 
with 16 legislators, 24 committee staff, and 17 legislative agency 
staff were conducted to analyze the strategies used to monitor recent 
educational reforms in 6 states — Arizona, California, Georgia, 
Florida, Minnesota, and Pennsylvania. Findings indicate that 
educational reform was conducted in ways that minimized time 
commitments and maximized political benefits for legislators. 
Legislators, or policymakers, preferred to utilize fire alarm 
techniques, and practicing evaluators tended to patrol. The 
recommendation is made for practicing evaluators to utilize fire 
alarm strategies in order to operat*? as efficiently as their 
legislative counterparts. (7 references) (LMI) 
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Oversight is the evaluative function of legislatures. 

During oversight, legislators monitor policy implementation to 
detect and remedy problems, or violations of legislative intent. 
Most oversight, like formative evaluation, is directed at 
producing information that can be used by legislators and 
administrators to improve policies during implementation. 
Indeed, legislators seldom use oversight to decide whether a 
policy should be discontinued. 

This article examines how state legislatures practice 
evaluation by analyzing oversight strategies used to monitor 
recent education reforms in six states (Arizona, California, 
Florida, Georgia, Minnesota and Pennsylvania).^ The central 
theme of the article is the inextricable link between politics 
and oversight: the choice of oversight strategy, and therefore, 
the methods legislatures use to evaluate policy implementation, 
are driven largely by politics. The study found that oversight 
of education reform in the six sample states was conducted in 
ways that minimized time commitments and maximized political 
benefits for legislators. 

FIRE-ALARM VERSUS POLICE-PATROL OVERSIGHT 

One way to analyze evaluation practice in state legislature 
is with the conceptual framework of dual oversight strategies 
developed by McCubbins and Schwartz in their 1984 study of 
Congress. "Fire-alarm" oversight involves selective monitoring, 
triggered by complaints from citizens and interest groups who 
bring potential problems to legislators' attention. The other 
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strategy, "police-patrol" oversight, is more active, direct and 
centralized: legislators monitor policies to detect problems 
during implementation (as opposed to responding reactively to 
alarms from outsiders) . Sunset reviews and statutory evaluation 
requirements, such as legislatively-mandated reports, are police- 
patrol techniques. Examples of fire-alarm technic[ues are 
constituent-service (casework) activities and, what Lois-Ellin 
Datta terms, GAO's "cop-on-the-beat" studies, which are in 
response to highly specific allegations when something has gone 
wrong. Committee hearings can be either type, depending upon 
whether the hearing was called to patrol for problems, or whether 
it was called in response to an alarm signalling a potential 
problem. 

A comparison of the two oversight strategies suggests 
different costs and benefits. Police-patrol oversight is more 
labor intensive and time-consuming. Legislatures must establish 
special procedures for patrolling, which usually are a drain on 
members' time. And, inevitably legislators spend a great deal of 
time examining successful policies where implementation meets 
legislative intent and is effective. 

On the other hand, police-patrol oversight can be an 

effective technique for deterring violations of legislative 

intent, as one person interviewed for this study explained: 

When the committee commissioned Price Waterhouse to conduct 
a management study of the state department of education, it 
was like sending in the cops, and it had a good deterrent on 
"weird" behavior. 

The department knew the cops were coming in and they cleaned 
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up their act. I received lots of calls from department 
people saying that now that Price Waterhouse is coming in, 
I*m getting heat from my boss to do things the right way. 

The expectation in the present study was that the police- 
patrol strategy would be used in California, Florida and 
Pennsylvania where legislatures have greater resources for 

evaluation large staffs, long legislative sessions, and many 

full-time legislators. It also was expected that state 
legislatures would patrol to embarrass or harass the implementing 
agency when relations between the legislature and the executive 
were strained, as when different political parties controlled 
each branch. 

McCubbins and Schwartz conclude in their study of Congress 

that fire-alarm oversight is the preferred form, because it is 

more efficient for evaluating and improving policy 

implementation. The strategy relies on someone other than 

legislators for surveillance, and legislators get involved only 

after an alarm is sounded when a potential problem already is 

identified. According to the researchers: 

...a fire-alarm policy enables congressmen to spend less 
time on oversight, leaving more time for other profitable 
activities, or to spend the same time on more personally 

profitable oversight activities on addressing complaints 

by potential supporters. Justly or unjustly, time spent 
putting out visible fires gains one more credit than the 
same time spent sniffing for smoke (1984, p. 168). 

In view of these benefits, it was expected that fire-alarm 

oversight would be the predominant strategy in the six sample 

states, as it is in Congress. It also was expected that because 

alarms point out problems, legislators would rely on the strategy 
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to direct their evaluations of the large, omnibus (multi-faceted) 
reforms of California and Georgia. 

FZNDZNOS 

The findings presented here are based on fifty-seven elite 
interviews, conducted with legislators (16), committee staff 
(24) , and legislative agency staff (17) who are responsible for 
overseeing education reforms enacted in the previous six years. 
Elite interviewing, compared to structured survey instruments, 
treats each interviewee in an individualized, nonstandard way by 
giving the researcher flexibility in asking questions and the 
interviewee or "elite" more flexibility in responding (Dexter, 
1970) . The interview topics for this study focused on 
institutional and political constraints on oversight, resources 
available to legislatures for oversight and the extent to which 
they were used, and motivations of legislators in monitoring 
contracts for education reform. 

Every state in the sample had at least some education reform 
policies evaluated with the fire-alarm strategy. By contrast, 
the police-patrol technique was used with reforms in only three 

states Arizona, Florida and Minnesota. Data from the 

interviews suggest that legislators in the six sample states 
practiced evaluation in ways that helped them achieve their 
political goals, such as serving constituents. Most legislators 
preferred responding to alarms, and when patrolling techniq'^es 
were used, legislators integrated political benefits into their 
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patrol strategies. The sections that follow provide evidence for 
these conclusions. 
When Legislators Patrol 

As expected, legislatures in the sample with a history of 
less than cooperative relations with the executive branch 
incorporated patrolling provisions into education reform 
legislation. The Arizona and Florida legislatures set a priori 
terms in the legislation to facilitate surveillance of policy 
implementors, with little role for state departments of 
education. Evaluation reports by independent contractors we*.e 
mandated and special oversight units were established with the 
intent that legislators would patrol for violations during 
implementation. 

A major education reform in Arizona was the teacher career 
ladder demonstration program which restructures the compensation 
and promotion options of teachers by rewarding them on the basis 
of performance, including student academic progress. In 
authorizing the program, the legislature created a joint 
legislative committee to implement as well as monitor the career 
ladder program, largely because the legislature was dissatisfied 
with the department of education's management of earlier reforms. 
Each member of the joint committee was responsible for visiting 
several local districts to monitor i^nplementation. As a result, 
members of the committee had public relations opportunities, and 
the joint committee (rather than the department) had control over 
information about implementation, which, of course, enhanced the 
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legislature's ability to patrol for violations. 

Legislative monitors discovered early during implementation 
of the career ladder program that some districts, contrary to 
intent, continued to use traditional salary schedules, which are 
based on education and experience, for teacher compensation 
decisions. The joint committee called a meeting with 
representatives of local districts, and a revised policy 
statement stressing performance-based schedules subsec[uently was 
issued to clarify legislative intent. Remedying the violation 
was speedy. In addition, with legislators serving as program 
field monitors, there was the political boon for committee 
members of frequent constituency contact. 

The Florida legislature, which has a similar history of poor 
relations with the department of education, established, in its 
1983 education reform legislation, the Florida Quality 
Instructional Incentives Council to oversee implementation of the 
reforms. The council members included legislators as well as 
representatives of the business community and, according to 
council staff, shared responsibility probably reduced the time 
most legislators must devote to oversight. Reporting 
requirements, like the ones written into Arizona's career ladder 
legislation, stipulated when evaluations were due, that 
independent evaluators were required, and the type of information 
that reports should include. Florida's state department of 
education served only a minor role as contract monitor. (In 
Arizona the legislature completely bypassed the department by 
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appointing Northern Arizona University as the independent 
evaluator.) A commitment to good public policy initially may 
have motivated legislators to volunteer for the oversight 
committees in Arizona and Florida, but interest was sustained 
because members were able to do something to benefit their 
districts and for which they could claim credit. The special 
oversight mission of these committees, in effect, gave 
legislators political benefits for monitoring. 

Contrary to expectations, patrolling was conducted in states 
without regard to legislative resources. The Florida 
legislature, which has a sizeable staff and long sessions, 
evaluated by patrolling, however, the two other high capacity 
states, California and Pennsylvania, did not. Conversely, 
Arizona with a low capacity legislature used the patrolling 
technique even though it overtaxed the legislators who served as 
program monitors and also legislative staff 

Another unanticipated finding was the key oversight role 
played by the sponsors or "legislative champions" of the 
education reforms. Equipped with staff resources (all held 
leadership positions) , the champions spent considerable time 
patrolling for violations by keeping in close communication with 
policy implementors through frequent meetings, telephone calls 
and occasional hearings. 

Interest in oversight among legislative champions stemmed 
partly from tne champions* strong ideological commitment to the 
education reform. Oversight was used to improve policy and to 
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protect against opposition. The representative from Minnesota 
who championed the Post secondary Enrollment Options Act (PSEO) , 
which gives 11th and 12th graders the option of attending 
postsecondary institutions, patrolled during implementation to 
identify provisions that needed refinement. At the same time, 
groups opposing the reform, such as school districts that feared 
severe financial losses if many high school students and the 
money to educate them were transferred to postsecondary 
institutions, were on the lookout for problems that might justify 
the law's repeal. Based on information gleaned from monitoring, 
the champion quickly introduced amendments to rectify PSEO's 
problems: 

Oversight was a political strategy to ensure I had 
control of the changes. I also felt that if I didn't 
fix the administrative problems soon, amendments would 
pass that would repeal the program. 

Some of the changes included deadlines that required students to 

notify districts the spring before the next academic year of 

their intent to ^larticipate, and counseling provisions to help 

students make more thoughtful decisions. 

The impetus for oversight also came from the personal 

benefits of oversight, as a staff member to one of the 

legislative champions observes: "Legislators need to protect 

their reputations, and not following up on something can be worse 

than failing to move on to the next issue. The reforms are his 

legacy." The education reforms to their champions were vehicles 

for building reputations in the legislature and with 
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constituents, providing nothing disastrous happened during 
implementation . 

In sum, patrolling generally was practiced by a specialized 
few, either specially-created oversight units or legislative 
champions, when the education reforms were priorities on 
committee agendas or when the reforms furthered the personal 
goals of legislators. The effect was increased efficiency as 
selected legislators became specialists in monitoring education 
reforms. To enhance the appeal of monitoring, opportunities for 
improving legislators' reputations and reelection prospects often 
were integrated into oversight strategies. 
Legislative Preference fo r Alarms 

Notwithstanding the willingness of legislative champions and 
special committees to patrol for violations, responding to alarms 
was the predominant form of oversight used by legislatures with 
respect to the education reforms. Oversight was selective, 
usually triggered by complaints from constituents, who served as 
volunteer monitors, augmenting the resources of part-time 
legislatures. Legislatures spent less time on oversight because 
constituent volunteers generally assumed the time-consuming task 
of monitoring implementation. By waiting for alarms to sound, 
legislators also increased the political benefits of oversight: 
members received credit from constituents for intervening to 
investigate the cause of their complaints (McCubbins and 
Schwartz, 1984). 

In Pennsylvania, where the state board of education is 
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allowed to make law, the legislature serves as a **court of 
appeals" for disgruntled constituents and interest groups. 
Teacher unions were angry when the board adopted a rule requiring 
teachers with Masters degrees to take six credits of continuing 
education every five years. The unions appealed to the 
legislature, arguing that board policy harmed their constituents 
and ultimately the rule was statutorily repealed. The Senate 
Education Committee, which led the fight, knew it would be 
difficult to garner legislative support for an education bill 
with specific program provisions, so they waited until June when 
the omnibus education bill was considered and added the appeal. 
In responding to the alarm, the senators on the education 
committee won credit from teacher unions, and, equally important, 
they avoided having to cash in the political chits that a 
separate bill likely would have required. 

In California alarms were sounded when the department of 
education began to implement an education reform statewide 
instead of on a pilot basis as intended by the legislature, 
apparently because selecting pilot districts was unappealing for 
political reasons. Education lobbyists also charged that local 
implementation was contrary to the spirit of the law: some 
districts reclassified twelfth graders as eleventh graders to 
improve school scores. Education committee members briefly 
considered terminating the program, but ultimately opted for a 
milder control, using fiscal sanctions to reaffirm legislative 
intent. The action soothed members' consciences but, perhaps 
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most important, all school districts continued to receive money 
and legislators lost little politically with their constituents. 

Georgia's Quality Basic Education Act, similar to 
California's major reform act of 1983 (Senate Bill 813), is an 
educational milestone: it is large, comprehensive, and backed by 
a substantial investment of state money. The bill was rushed 
through the legislature in 1985 and was closely tracked during 
implementation by education interest groups whose attention was 
drawn by promises of increased funding. Special education groups 
voiced the loudest complaints. Apparently in the rush to write 
the bill, legislators miscalculated the weights in the funding 
formula for special education. The fire alarm led legislators to 
amend the law only one year after passage, thereby reassuring 
interest groups of the legislature's intent to increase funding 
for education. 

As expected in California and Georgia where reforms were 
enacted through omnibus legislation, legislators practiced only 
the fire-alarm strategy. According to committee staff in both 
states, monitoring all the provisions uould have been 
overwhelming. The alarms were useful for identifying the areas 
in greatest need of repair and also helped protect legislators 
from information overload. 

CONCLUSION 

The six states conducted oversight of education reform in 
ways that minimized time commitments and maximized political 
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benefits for legislators. When the police-patrol strategy was 

used, time demands on legislators were reduced by having 

legislative champions or special purpose units act on beha] f of 

the legislature, and there were political benefits for the 

overseers as well. However, judging from the frequency of use of 

the oversight strategies, the states preferred the fire-alarm 

strategy, and evaluations most often were selective, focusing on 

reports of possible violations from constituents and interest 

groups with whom legislators later claimed credit for serving. 

The preference for fire-alarm oversight is eminently 

rational in terms of what we already know about the use of 

evaluations in the legislative process. Responding to alarms, 

according to the interviewees, ensures that evaluation 

information will be: 

o Timely constituents who want action are sensitive 

to the legislative cycle and sound alarms when 
legislators are able to respond (Chelimsky, 1987) . 

o Relevant alarms deal with concerns important to 

constituents and focus on topics that legislators can 
do something about (Weiss & Weiss, 1981) . 

o From credible sources alarms are sounded by 

constituents, mainly organized interest groups whom 
legislative staff routinely listen to (Weiss, 1989) . 

o Oriented toward program improvement alarms provide 

formative information that identify possible problems 
and recommendations for changes (Alkin et al., 1974). 

For practicing evaluators who are concerned about enhancing 

decision-making, this study offers some important lessons. The 

evaluation literature has never been clear about which decisions 

evaluators should try to enhance. The results from this study 
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suggest policy makers are interested primarily in decisions that 
will put out fires and improve programs. But practicing 

evaluators more often do something quite different patrol. 

Resources for in-house evaluation shops, for example, 
traditionally are poured into what Huberman and Miles (1984) 

refer to as "monitoring for fidelity" evaluations is program 

implementation faithful to program design? or into more 

technically-oriented evaluations that feature test score 
analysis, for instance. Practicing evaluators may be trying to 
enhance decision-making, but the decisions their evaluations 
address may be of only secondary interest to policy makers. 

At the very least, the findings here suggest that practicing 
evaluators ought to spend more time fighting fires and conducting 
evaluations that enhance decisions about program improvement. 
In-house evaluation shops perhaps ought to be recreated as fire 
departments of evaluators with the capability to ferret out facts 
and put out fires. Legislators evaluate in ways that maximize 
benefits and minimize time commitments, practicing evaluators 
should be similarly efficient in their activities. 
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NOTES 

1. In lina with national trends, education reforms in the sample 
states focused on raising academic standards for students and 
improving how teachers are trained, recruited and compensated. 
Reforms increased course rec[uirements for graduation (Arizona, 
California, Florida, Georgia, Pennsylvania) ; expanded student 
testing programs (Arizona, California, Georgia, Pennsylvania); and 
lengthened the school day (California, Florida). Minnesota's major 
reform al loved eleventh and twelfth graders the choice of attending 
high school or a postsecondary institution such as a community 
college or vocational center. Reforms directed at improving 
teacher quality stiffened teacher certification requirements 
(Georgia, Pennsylvania) or moved toward performance-based 
compensation systems (Arizona, California, Florida). 



2. Arizona's Senate Education Committee only has two staff 
members. The senior staffer doubled as project director for the 
career ladder program and staff to the joint committee, in addition 
to working for the education committee. 
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